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Abstract — We consider lossy compression of a binary symmet- 
ric source by means of a low-density generator-matrix code. We 
derive two lower bounds on the rate distortion function which 
are valid for any low-density generator-matrix code with a given 
node degree distribution L{x) on the set of generators and for 
any encoding algorithm. These bounds show that, due to the 
sparseness of the code, the performance is strictly bounded away 
from the Shannon rate-distortion function. In this sense, our 
bounds represent a natural generalization of Gallager's bound on 
the maximum rate at which low-density parity-check codes can 
be used for reliable transmission. Our bounds are similar in spirit 
to the technique recently developed by Dimakis, Wainwright, and 
Ramchandran, but they apply to individual codes. 

L Introduction 

We consider lossy compression of a binary symmetric 
source (BSS) using a low-density generator-matrix (LDGM) 
code as shown in Figure [T] More precisely, let S G F™ 
represent the binary source of length m. We have S = 
{^1, S'2, . . . , Sm}, where the {S'i}™ ^ are iid random variables 
with F{Si = 1} = i, i e [to]. Let S denote the set of all 
source words. 
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Fig. 1. The Tanner graph corresponding to a simple LDGM code used for 
lossy compression of a BSS. We have m = 7, /? = 4, and L{x) = x^. 




Given a source word s £ 5, we compress it by mapping it to 
one of the 2™^ index words w G W = F™^, where R is the 
rate, R G [0, 1]. We denote this encoding map hy i : s W 
(the map can be random). The reconstruction is done via an 
LDGM code determined by a sparse binary mR x to generator 
matrix G. Let s denote the reconstructed word associated to w. 
We have s = wG. We denote this decoding map hy g : w 1-^ s. 
Let S denote the code, S = {s'-^\ . . . , s'-^"'"^}, £ F™. 
The codewords are not necessarily distinct. 

We call the components of the index word w = 
{wi, . . . , w,nR} the generators and the associated nodes in the 
factor graph representing the LDGM code the generator nodes. 
We assume that these generators nodes have a normalized 



degree distribution L{x) = This means that Li 

represents the fraction (out of mR) of generator nodes of 
degree i. 

We are interested in the trade-off between rate and distortion 
which is achievable in this setting. Let d(-, •) denote the 



Hamming distortion function, d 
distortion is then given by 
1 



f:?' X FJj 



N. The average 



-E[d{S,g{f{S))]. 
m 

We are interested in the minimum of this average distortion, 
where the minimum is taken over all LDGM codes of a given 
rate, generator degree distribution L{x), and length, as well 
as over all encoding functions. 

n. Review 

Given the success of sparse graph codes applied to the 
channel coding problem, it is not surprising that there is also 
interest in the use of sparse graph codes for the source coding 
problem. Martinian and Yedidia [1] were probably the first to 
work on lossy compression using sparse graph codes. They 
considered a memoryless ternary source with erasures and 
demonstrated a duality result between compression of this 
source and the transmission problem over a binary erasure 
channel (both using iterative encoding/decoding). Mezard, 
Zecchina, and Ciliberti [2] considered the lossy compression 
of the BSS using LDGM codes with a Poisson distribu- 
tion on the generators. They derived the one-step replica 
symmetry -breaking (IRSB) solution and the average rate- 
distortion function. According to this analysis, this ensemble 
approaches the Shannon rate-distortion curve exponentially 
fast in the average degree. They observed that the iterative 
interpretation associated to the IRSB analysis gives rise to 
an algorithm, which they called survey propagation. In [3] 
the same authors implement an encoder that utilizes a Tanner 
graph with random non-linear functions at the check nodes 
and a survey propagation based decimation algorithm for data 
compression of the BSS. In [4], Wainwright and Maneva also 
considered the lossy compression of a BSS using an LDGM 
code with a given degree distribution. They showed how 
survey propagation can be interpreted as belief propagation 
algorithm (as did Braunstein and Zecchina [5]) on an enlarged 
set of assignments and demonstrated that the survey propa- 
gation algorithm is a practical and efficient encoding scheme. 
Recently, Filler and Friedrich [6] demonstrated experimentally 
that even standard belief propagation based decimation algo- 
rithms using optimized degree distributions for LDGM codes 



and a proper initialization of the messages can achieve a rate- 
distortion trade-off very close to the Shannon bound. Martinian 
and Wainwright [7], [8], [9] constructed compound LDPC 
and LDGM code ensembles and gave rigorous upper bounds 
on their distortion performance. A standard LDGM code 
ensemble is a special case of their construction, hence they also 
provide upper bounds on the rate-distortion function of LDGM 
ensembles. By using the first and second moment method 
they proved that a code chosen randomly from the compound 
ensemble under optimal encoding and decoding achieves the 
Shannon rate-distortion curve with high probability. Finally, 
they pointed out that such constructions are useful also in 
a more general context (e.g., the Wyner-Ziv or the Gelfand- 
Pinsker problem). Dimakis et al [10] were the first authors to 
provide rigorous lower bounds on the rate-distortion function 
of LDGM code ensembles. 

Theorem 1 (Dimakis, Wainwright, Ramchandran [10]): 
Let iS be a binary code of blocklength m and rate R chosen 
uniformly at random from an ensemble of left Poisson LDGM 
Codes with check-node degree r. Suppose that we perform 
MAP decoding. With high probability the rate-distortion pair 
(R, D) achieved by S fulfills 

1 - h(D) , , , 



R > 



1 

A. Outline 

In the spirit of Gallager's information theoretic bound for 
LDPC codes, we are interested in deriving lower bounds on 
the rate-distortion function which are valid for ayiy LDGM 
code with a given generator node degree distribution L{x). 
Our approach is very simple. Pick a parameter D, D G [0, |] 
(think of this parameter as the distortion). Consider the set of 
"covered" sequences 



C{D) = IJ B{s,Dm), 



(1) 



ses 



where B{x, i), x e F™, i G [m], is the Hamming ball of radius 
i centered at x. In words, C{D) represents the set of all those 
source sequences that are within Hamming distance at most 
Dm from at least one code word. 

Recall that for any s e 5, f(s) G W represents the index 
word and that g(f (s)) denotes the reconstructed word. We have 



d(s,g(f(s)))> 



0, s G C{D), 
Dm, s<=:W^\C{D). 



Therefore, 

-E[d(5,g(f(5)))] 

TO 

= - E 2- d(., g(f (.))) > — g(f (.))) 

sGFJ' seV^\C{D) 

> 2-"''D\¥^'\CiD)\ > D(l -2-™|C(L>)|). (2) 

If the codewords are well spread out then we know from 
Shannon's random coding argument that for a choice D = 
h-\l - R), \C{D)\ « 2", [11]. But the codewords of an 



LDGM code are clustered since changing a single generator 
symbol only changes a constant number of symbols in the 
codeword. There is therefore substantial overlap of the balls. 
We will show that there exists a D which is strictly larger 
than the distortion corresponding to Shannon's rate-distortion 
bound so that |C(-D)| is exponentially small compared to 2™ 
regardless of the specific code. From (|2]i this implies that the 
distortion is at least D. 

To derive the required upper bound on |C(£')| we use two 
different techniques. In Section Hill we use a simple combina- 
torial argument. In Section IIVI on the other hand, we employ 
a probabilistic argument based on the "test channel" which is 
typically used to show the achievability of the Shannon rate- 
distortion function. 

Although both bounds prove that the rate-distortion function 
is strictly bounded away from the Shannon rate-distortion 
function for the whole range of rates and any LDGM code, 
we conjecture that a stronger bound is valid. We pose our 
conjecture as an open problem in Section [Vl 

III. Bound Via Counting 

Theorem 2 (Bound Via Counting): Let S be an LDGM 
code with blocklength to and with generator node degree 
distribution L{x) and define L' = L'{1). Let 



/(^•) - 11(1 



-x')^\ a(x) 



i=0 



R{x) = -Ai±|f, D{x) = -—-a{x)R{x). 

1 - log ^ ^ + ^ 

For R G [-J7, 1] let x{R) be the unique positive solution of 
R{x) = R. Define the curve Di^R) as 



D{x{R)),RG[^,l]. 

Then, for any blocklength to, the achievable distortion of an 
LDGM code of rate R and generator degree distribution L{x) 
is lower bounded by D{R). 

Discussion: (i) As stated above, if we are considering a single 
code of rate R then the lower bound on the distortion is D{R). 
If, on the other hand we are considering a family of codes, 
all with the same generator degree distribution L{x) but with 
different rates R, then it is more convenient to plot the lower 
bound in a parametric form. First plot the curve {D{x), R{x)) 
for X G [0,1]. Then connect the point {D = ^,R — 0) 
to the point on the {D{x), R{x)) curve with R{x) = jj 
by a straight line. The resulting upper envelope gives the 
stated lower bound for the whole range. This construction is 
shown in Figure |2] (ii) Although this is difficult to glance 
from the expressions, we will see in the proof that for any 
bounded generator degree distribution L{x) the performance 
is strictly bounded away from the Shannon rate-distortion 
function. From a practical perspective however the gap to the 
rate-distortion bound decreases quickly in the degree. 




Fig. 2. Constniction of the bound for codes with L(x) = so that L' = 2 
(all generator nodes have degree 2). The solid gray curve corresponds to the 
Shannon rate-distortion curve. The black curve just above, which is partially 
solid and partially dotted, corresponds to the curve {D{x), R{x)) for x G 
[0, 1]. It starts at the point (0, 1) (which corresponds to a; = 0) and ends at 
( ^2L'^ ~ \' (L'Y^ ~ 3) which corresponds to a; = 1. The straight line goes 
from the point {D{x{jj)), -jj) to the point (5,0). Any achievable (R, D) 
pair must lie in the lightly shaded region. This region is strictly bounded away 
from the Shannon rate-distortion function over the whole range. 



Example 1 (Generator-Regular LDGM Codes): Consider 
codes with generator degree equal to 1 and an arbitrary 
degree distribution on the check nodes. In this case we have 



1 



and a{x) 



Figure [3] compares the 



lower bound to the rate-distortion curve for 1 = 1, 2, and 3. 
For each case the achievable region is strictly bounded away 
from the Shannon rate-distortion curve. 
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and 3. For 1 : 
, and R = 



2 the 3 gray 
r respectively. 



Fig. 3. Bounds for L{x) = X''- for 1 = 1, 2, 
dots correspond to the special cases R = ^, R = ^ , 
The corresponding lower bounds on the distortion are D{^) > 0.0616 > 
0.0614905 (rate-distortion bound), Z)(i) > 0.115 > 0.11 (rate-distortion 
bound), and > 0.1924 > 0.1461 (rate-distortion bound). 



Example 2 ({l,r)-Regular LDGM Codes): In this case we 
have R = l/r and L{x) = x-"-. The same bound as in 
Example [T] applies. The three special cases (l = 2, r = 3), 
(1 = 2,r = 4), and (l = 2, r = 5), which correspond to 
R = |, i? = i, and i? = | respectively, are marked in 
Figure |3] as gray dots. 

Example 3 (r -Regular LDGM Codes of Rate R): Assume 
that all check nodes have degree r and that the connections 
are chosen uniformly at random with repetitions. For large 
blocklengths this implies that the degree distribution on the 



variable nodes converges to a Poisson distribution, i.e., we 
have in the limit 



Let us evaluate our bound for this generator degree distribu- 
tion. Note that since the average degree of the check nodes is 
fixed we have a different generator degree distribution L{x) 
for each rate R. Figure |4] compares the resulting bound with 
the Shannon rate-distortion function as well as the bound 
of Theorem [T] The new bound is slightly tighter. But more 
importantly, it applies to any LDGM code. 
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Fig. 4. Lower bound on achievable {R, D) pairs for r-regular LDGM codes 
with a Poisson generator degree distribution and r = 2,4. The dashed curve 
con'esponds to the bound of Theorem [T] and the solid black curve represents 
the bound of Theorem |2] The gray curve is the Shannon rate-distortion 
tradeoff. 

Proof of Theorem |2] From the statement in Theorem |2] 
you see that the bound consists of a portion of the curve 
{D{x),R{x)) and a straight-line portion. The straight-line 
portion is easily explained. Assume that all generator nodes 
have degree 1 (for the general case replace all mentions of 
1 by the average degree L'). Then the maximum number of 
check nodes that can depend on the choice of generator nodes 
is nl. Therefore, if the rate R is lower than j then at least 
a fraction (1 — of the check nodes cannot be connected 
to any generator node. For those nodes the average distortion 
is i, whereas for the fraction Rl of the check nodes which 
are (potentially) connected to at least one generator node the 
best achievable distortion is the same for any < i? < i. It 
suffices therefore to restrict our attention to rates in the range 
1] and to prove that their [R, D) pairs are lower bounded 
by the curve {D{x), R{x)). 

As a second simplification note that although the bound is 
valid for all blocklengths m we only need to prove it for the 
limit of infinite blocklengths. To see this, consider a particular 
code of blocklength m. Take k identical copies of this code 
and consider these k copies as one code of blocklength km. 
Clearly, this large code has the same rate R, the same generator 
degree distribution L{x), and the same distortion D as each 
component code. By letting k tend to infinity we can construct 
an arbitrarily large code of the same characteristics and apply 
the bound to this limit. Since our bound below is valid for 



any sequence of codes whose blocklength tends to infinity the 
claim follows. 

Pick w e N so that Dm + w < ^. Then 



\[jBis,Dm)\ 
1 



B{s,Dm + w)\ 



^) i2mRi2mh{D+w /m) 
ii^+R+h{D+a{x^)R)+o„,{l)) 



(iii) ^m{-Rlog 

To see (i) note that a "big" sphere B(s, Dm + w), where 
s G S, contains all "small" spheres of the form B{s', Dm), 
where s' g 5 so that d(s, s') < w. Let A„i{w) be the 
number of codewords of Hamming weight at most w. Then, 
by symmetry, each small sphere B{s', Dm) is in exactly 
Amiw) big spheres B{s, Dm + w). It follows that every point 
in [J.^g B{s, Dm) is counted at least A„i{w) times in the 
expression J^ses \B{s, Dm + w)\. 

Consider now step (ii). We need a lower bound on A,n{w). 
Assume at first that all generator nodes have degree 1. Assume 
that exactly g generator nodes are set to 1 and that all other 
nodes are set to 0. There are (™^^) ways of doing this. Now 
note that for each such constellation the weight of the resulting 
codeword is at most w = gl. It follows that in the generator 
regular case we have 

w/l 

I mit \ 



3=0 



We can rewrite Q in the form 



Am{w) > 



i=0 



_coef{(l + .T^)"-",a;'}, (4) 



where coef{(l + x-"^)™^, x'} indicates the coefficient of the 
polynomial (1 + x-"^)™^ in front of the monomial x'. The 
expression ^ stays valid also for irregular generator degree 
distributions L{x) if we replace (1 + a;-"-)'"^ with /(x)™^, 
where f{x) = Yiii^ + a;*)^* as defined in the statement of the 
theorem. This of course requires that n is chosen in such a 
way that nLj G N for all i. 

Define N,n{w) = J2'"^QCoef{f{x)"'^,x'}, so that dUi can 
be restated as Am{w) > Nm{w). Step (ii) now follows by 
using the asymptotic expansion of N,n{w) stated as Theorem 1 
[12], where we define uj = w/{mR) and where x^j is the 
unique positive solution to a{x) — ui. 

Finally, to see (iii) we replace w by mRa{xuj) and thus we 
get the claim. Since this bound is valid for any w S N so that 
Dm + w < ^ we get the bound 

^■log\C{D)\<g{D,R), 



lim 

rn — >OG m 



where 

giD,R) 



inf 

D+a{x)R<^ 



-R\og 



/(^) 

.j,a{x) 



R + h{D + a{x)R). 



Now note that as long as g{D,R) < 1, |C(£')| is exponen- 
tially small compared to 2™. Therefore, looking back at (|2]i 
we see that in this case the average distortion converges to at 
least D in the limit rn oo. We get the tightest bound by 
looking for the condition for equality, i.e. by looking at the 
equation g{R, £>) = 1. If we take the derivative with respect 
to x and set it to then we get the condition 

^D + Ra{x). 



1 + 

Recall that D + a{x)R < \, so that this translates to x < 1. 
This means that x < 1. Replace D + a{x)R in the entropy 
term by set the resulting expression for g{R, x) equal to 
1, and solve for R. This gives i? as a function of x and so we 
also get D as a function of x. We have 



R{x) 



D{x) 



1 



a{x)R{x) 



A check shows that x = Q corresponds to {D, R) — (0, 1) and 
that X = 1 corresponds to {D,R) ~ ( "^^^Z , ^j]yi ). Further, 



R and D are monotone functions of x. Recall that we are 
only interested in the bound for R e [j7jl]- We get the 
corresponding curve by letting x take values in [Q,x{jj)]. For 
smaller values of the rate we get the aforementioned straight- 
line bound. 

Looking at the above expression for g{D,R) one can see 
why this bound is strictly better than the rate-distortion curve 
for D G (0, i). Assume at first that the generator degree 
distribution is regular. Let the degree be 1. In this case a quick 
check shows that — i?log-^^ is equal to — Since 
a(0) = we get the rate distortion bound if we set x = 0. The 
claim follows by observing that a(x) is a continuous strictly 
increasing function and that h{x) has an infinite derivative 
at X = while h{D + a{x)R) has a finite derivative at 
X = 0. It follows that there exists a sufficiently small x so 
that Rh{^) is stricdy larger than h{D + a{x)R) - h{D) 
and so that D + a{x)R < i. Hence, g{D,R) is strictly 
decreasing as a function of x at .x = 0. This bounds the 
achievable distortion strictly away from the rate-distortion 
bound. The same argument applies to an irregular generator 
degree distribution; the simplest way to see this is to replace 
1 by the maximum degree of L(x). 

IV. Bound Via Test Channel 

Instead of using a combinatorial approach to bound |C(D)| 
one can also use a probabilistic argument using the "test 
channel" shown in Figure |5] 

For the cases we have checked the resulting bound is 
numerically identical to the bound of Theorem |2] (exclud- 
ing the straight-line portion). We restrict our exposition to 
the regular case. The generalization to the irregular case is 
straightforward. 

Theorem 3 (Bound Via Test Channel): Let S be an LDGM 
code with blocklength m, generator degree distribution 
L(x) = x-"^, and rate R. Then for any pair {R,D), where 




/ D' \d(s,s)+d(s,s') 

_ / /")' \ d(s,s)-\-w 

■u;=0 



w=0 
d(s' ,s)—w 



-D'J 

]J' \ Dm+w 



> EA«H2— (^) 



(1 -D')"" 
D' \>" 



Fig. 5. The generator words W are chosen uniformly at random from W. 
This generates a codeword S uniformly at random. Each component of S is 
then sent over a binary symmetric channel with transition probability D' . 



D is the average distortion, we have 



^ ^-,nR-mhiD)-,nKLiD\\D') ^ Amiw) (^-^'^ , 

■w=0 

where Am{w) denotes the number of codewords in S of 
Hamming weight w. Due to the linearity of the code this is also 
the number of codewords in S of Hamming distance w from 
s. Using summation by parts and setting c — D' / {l — D') < 1, 
we have 



R > sup 



1 - h{D) -KL{D\\D') 



> 



D<D'<^ 1 -log2(l 

1 - h{D) 
l-l0g2(l+(T^ 



{D'Y 
(1-D'}\ 

> l-h{D), 



w=0 



m w — 1 



^ A,n{w)c 



w=0 i=0 



where KL(i:>||7:>') = Dlog^iD/D'' 
D)/{1~D')). 

Proof. The same remark as in the proof of Theorem |2] applies: 
although the bound is valid for any blocklength it suffices to 
prove it for the limit of blocklengths tending to infinity. Also, 
for simplicity we have not stated the bound in its strengthened 
form which includes a straight-line portion. But the same 
technique that was applied in the proof of Theorem |2] applies 
also to the present case. 

As remarked earlier, the idea of the proof is based on 
bounding |C(-D)| by using the "test channel." More precisely, 
choose W uniformly at random from the set of all binary 
sequences of length mR. Subsequently compute S via S = 
WG, where G is the generator matrix of the LDGM code. 
Finally, let 5 = 5 + Z, where Z has iid components with 
P{Z, = 1} = D'. 

Consider the set of sequences s G C{D). For each such s 
we know that there exists an s G 5 so that d(s, J) < Dm. We 
have 

F{S^s\seC{D)} 

= ^ P{5 = s, S" = s' I s G C{D)} 
s'es 

m 

= E E ns = s,S = S'\seC{D)} 

'^^^ s'eS:d{s\s)^w 

= ^A,n{w)F{S^s,S^s' I s(EC{D),d{s\s) = w} 

™ / n' \d(s.s') 



w=Q i=0 ^ ^ 



^E^ (^^^c^"' + c"'^^ ^^mR _ /mi? 

■!ii=0 \ ^ / i=0 ^ * ■ 



w=0 



1 



c^- > -(1 + c^) 
w J m 



l\mft. 



The last step is valid as long as < i. In this case the 

maximum term (which appears at j^^m) is included in the 
sum (which goes to m/l) and is thus greater than equal to the 
average of all the terms, which is ^(l+c-"-)™^ . This condition 
is trivially fulfilled for Rl < 1. Assume for a moment that it 
is also fulfilled for Rl > 1 and the optimum choice of D'. It 
then follows that 

PIS' = s I s G C{D)} > i_2-'"(^+''(^)+'^(^ll^')--Ri°S2(i+c'))^ 
~ m 

Since 



seFJ* s£C{D) 

> |^(^)|i_2-™(fl+'»(£')+KL(r'||r>')-fl,iog2(i+c^))^ 



we have \C{D)\ < r„2'"(^+''(^)+KL(^ll^')-"i°82(i+c')). Pro- 
ceeding as in (|2]), we have 

E[d(S,g(f(S)))] >Z?(l-2-™|Cp)|) 

We conclude that if for some D < D' < I, R + h{D) + 
KL(D||D') - i?log2(l + tt^Xt) - 1 < then the distortion 



is at least D. All this is still conditioned on < 1 for 



the optimum choice of D' . For Rl < 1 we already checked 
this. So assume that Rl > 1. The above condition can then 



equivalently be written as D' < 



On the other 



i+(m-i)T 

hand, taking the derivative of our final expression on the rate- 
distortion function with respect to D' we get the condition for 
the maximum to be D' = r < r- We 



i+(m-i) T 



see therefore that our assumption < 1 is also correct in 
the case -Rl > 1. 

Numerical experiments show that the present bound yields 
for the regular case identical results as plotting the curve 
corresponding to g{D, R) ~ 1, where g{D, R) was defined 
in the proof of Theorem |2] This can be interpreted as follows. 
Choose D' equal to the optimal radius of the Hamming ball 
in the proof of Theorem |2] Then the points s' that contribute 
most to the probability of 5 = s must be those that have a 
distance to s of m{D' — D). 



V. Discussion and Open Questions 

In the preceding sections we gave two bounds. Both of them 
are based on the idea of counting the number of points that 
are "covered" by spheres centered around the codewords of an 
LDGM code. In the first case we derived a bound by double 
counting this number In the second case we derived a bound 
by looking at a probabilistic model using the test channel. 

An interesting open question is to determine the exact 
relationship of the test channel model to the rate-distortion 
problem. More precisely, it is tempting to conjecture that a pair 
[R, D) is only achievable if H{S) = m in this test channel 
model. This would require to show that only elements of the 
typical set of S under the test channel model are covered, 
i.e., have code words within distance D. For the test channel 
model it is very easy to determine a criterion in the spirit of 
Gallager's original bound. We have 



H{S) = H{W) + H{S I W) - H{W I S) 



niR 



iR + mh{D) - J2 H{Wg I ^, VFi, . . . , Wg.i) 



(i) 



9=1 
mR 



<mR + mh{D) - ^ H{Wg \ S, W^g) 

9=1 
mR 

(ii) 



m 



R + m.h{D) - H{Wg I Sg, W^g), 

9=1 



the resulting bound would read 

1 - h{D) 



R > 



• D 



This "bound" is similar in spirit to the original bound given 
by Gallager, except that in Gallager's original bound for 
LDPC codes we have a term corresponding to the entropy 
of single-parity check codes, whereas here we have terms that 
correspond to the entropy of repetition codes; this would be 
quite fitting given the duality of the problems. 
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where 5*3 denotes the subset of the components of the S 
vectors which are connected to the generator g. Step (i) follows 
since conditioning decreases entropy. Step (ii) follows since 
knowing (Sg,Wr^g), Wg is not dependent on S^g. The term 
H{Wg I Sg, Wr^g) rcprcscnts the EXIT function of a repetition 
code when transmitting over BSC(£') channel. If one could 
show that H{S) = to is a necessary condition for achieving 
average distortion of D then a quick calculation shows that 



